Every Data Science project starts with … ETL

• Have you ever written an R script to be run in a non-interactive way?
• Have you ever scheduled an R script to run without human intervention?
• Did it work?
• Do you have any R script running in production?
• … with central logging, proper monitoring and alerting?
• … run on cheap spot-instances with real-time performance metrics collected and feeding an AI picking the optimal instance type for the next run?

Every BI consulting firm have developed its own …

• job scheduler
• data repository and metadata documentation tool
• central logging
• and a few other things …

Every BI consulting firm have developed its own …

• job scheduler
• data repository and metadata documentation tool
• central logging
• and a few other things …

Today we can just pick the right open-source tool, such as

• Jenkins
• Airflow or Luigi
• cronR and taskscheduleR R packages
• cloud services (eg AWS Batch, Cloudwatch, Lambda)

Source: thecodinglove.com

Code style

kid driving the car animgif?

Code style

It can be anything … just be consistent!

• Google’s R Style Guide
• Hadley Wickham: Style guide. In Advanced R
• The tidyverse style guide
• Internal company style Guide

DRY

metrics[, ctr := round(clicks / impressions, 2)]
metrics[, rpc := round(revenue / clicks, 2)]

metrics[, ctr := clicks / impressions]
metrics[, rpc := revenue / clicks]
for (v in c('ctr', 'rpc')) {
  metrics[, (v) := round(get(v))]
}

for (v in names(which(sapply(metrics, class) == 'numeric'))) {
  metrics[, (v) := round(get(v))]
}

round_metrics <- function(data) {
    ...
}
metrics <- round_metrics(metrics)

library(myPackage)
metrics <- round_metrics(metrics)

Document and potentially open-source your code

… use version control!

git 101

Start from scratch:

git init .
git add .
git commit -m "hello world"
git push origin master

Contribute to an existing project:

git clone https://github.com/foo/bar
git branch -b feature-branch
git add .; git add -u .
git commit -m "add my new cool feature"
git push origin feature-branch

logger: a lightweight, modern and flexibly logging utility for R

library(logger)
log_threshold(TRACE)
log_info('Script starting up...')
#> INFO [2019-04-04 04:25:17] Script starting up...
log_info('The mtcars dataset has {nrow(mtcars)} rows!')
#> INFO [2019-04-04 04:25:17] The mtcars dataset has 32 rows!
log_eval(nrow(mtcars))
#> TRACE [2019-04-04 04:25:17] 'nrow(mtcars)' => '32L'

cat demo.R

library(logger)
pkgs <- available.packages()
log_info('There are {nrow(pkgs)} R packages hosted on CRAN!')
for (letter in letters) {
    lpkgs <- sum(grepl(letter, pkgs[, 'Package'], ignore.case = TRUE))
    log_level(if (lpkgs < 5000) TRACE else DEBUG,
              '{lpkgs} R packages including the {shQuote(letter)} letter')
}

INFO [2018-20-11 22:49:37] There are 13433 R packages hosted on CRAN!
DEBUG [2018-20-11 22:49:38] 6300 R packages including the 'a' letter
DEBUG [2018-20-11 22:49:38] 6772 R packages including the 'e' letter
DEBUG [2018-20-11 22:49:38] 5412 R packages including the 'i' letter
DEBUG [2018-20-11 22:49:38] 7014 R packages including the 'r' letter
DEBUG [2018-20-11 22:49:38] 6402 R packages including the 's' letter
DEBUG [2018-20-11 22:49:38] 5864 R packages including the 't' letter

Colorized output

library(logger)
log_layout(layout_glue_colors)
log_threshold(TRACE)
log_info('Starting the script...')
log_debug('This is the second log line')
log_trace('Note that the 2nd line is being placed right after the 1st one.')
log_success('Doing pretty well so far!')
log_warn('But beware, as some errors might come :/')
log_error('This is a problem')
log_debug('Note that getting an error is usually bad')
log_error('This is another problem')
log_fatal('The last problem')

The Anatomy of a Log Request

Custom formatter / layout / appender

log_threshold(WARN)
log_formatter(formatter_sprintf)
log_layout(layout_json(c('time', 'node', 'user', 'pid', 'msg')))
t <- tempfile()
log_appender(appender_file(t))
log_debug('I am a low level log message that will not be printed with a high log level threshold')
log_warn(
    'I am %s higher level log message that is very likely to be printed!',
    letters[1])
cat(readLines(t))

{
    "time": "2019-04-04 04:51:54",
    "node": "nevermind",
    "user": "daroczig",
    "pid": 22842,
    "msg": "I am a higher level log message that is very likely to be printed!"
}

More about logging

with_log_threshold(log_warn('booo'), threshold = ERROR)

log_with_separator('This message is worth a {1e3} words')
#> INFO [2019-03-29 19:00:15] =====================================================
#> INFO [2019-03-29 19:00:15] = This message is worth a 1000 words                =
#> INFO [2019-03-29 19:00:15] =====================================================

log_with_separator('Some critical KPI down!!!', separator = '$')
#> INFO [2019-03-29 19:00:15] $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
#> INFO [2019-03-29 19:00:15] $ Some critical KPI down!!!                         $
#> INFO [2019-03-29 19:00:15] $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

log_with_separator(paste(
  'A very important message with a bunch of extra words that will',
  'eventually wrap into a multi-line message for our quite nice demo :wow:'),
  width = 60)
#> INFO [2019-03-29 19:00:15] =================================
#> INFO [2019-03-29 19:00:15] = A very important message      =
#> INFO [2019-03-29 19:00:15] = with a bunch of extra words   =
#> INFO [2019-03-29 19:00:15] = that will eventually wrap     =
#> INFO [2019-03-29 19:00:15] = into a multi-line message     =
#> INFO [2019-03-29 19:00:15] = for our quite nice demo       =
#> INFO [2019-03-29 19:00:15] = :wow:                         =
#> INFO [2019-03-29 19:00:15] =================================

https://daroczig.github.io/logger

Why secure database credentials?

“When I woke up the next morning, I had four emails and a missed phone call from Amazon AWS – something about 140 servers running on my AWS account, mining Bitcoin.” – Andrew Hoffman

SELECT
  TO_HEX(CODE_POINTS_TO_BYTES([0xac, num2, num3, num4])) AS nonce
FROM
  UNNEST(GENERATE_ARRAY(0, 255)) num2,
  UNNEST(GENERATE_ARRAY(0, 255)) num3,
  UNNEST(GENERATE_ARRAY(0, 255)) num4
WHERE
TO_HEX(REVERSE(SHA256(SHA256(CONCAT(FROM_HEX(
  '000000204a4ef98461ee26898076e6a2cfc7c764d02b5f8d670832000000000000000000f99f5c4d5025979fcb33d245536a55b628d4564c075c0210cbbc941ad79fdbc5e491b55a494a5117'), CODE_POINTS_TO_BYTES([0xac, num2, num3, num4]) ))))) LIKE '000000000000000000%'

Loading MySQL configuration from global options

library(DBI)
con <- dbConnect(
    RMySQL::MySQL(),
    dbname = "shinydemo",
    host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
    username = getOption("my_username"),
    password = getOption("my_password"))

$ cat .Rprofile
options('my_username' = 'guest')
options('my_password' = 'guest')

Loading MySQL configuration from the keyring

con <- DBI::dbConnect(
    conf$drv,
    dbname = conf$dbname,
    host = conf$host,
    username = keyring::key_get("my_username"),
    password = keyring::key_get("my_password"))

Using a pre-configured Data Source Name

con <- DBI::dbConnect(odbc::odbc(), dsn = "shinydemo")

Using a MySQL configuration file

con <- dbConnect(RMySQL::MySQL(), group = "shinydemo")

[shinydemo]
user=guest
password=guest
database=shinydemo
host=shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com

Loading MySQL configuration from more general, custom files

mysql_user <- readRDS('secrets/mysql_user.RData')
mysql_pass <- readRDS('secrets/mysql_pass.RData')

library(DBI)
con <- dbConnect(
    RMySQL::MySQL(),
    dbname = "shinydemo",
    host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
    username = mysql_user,
    password = mysql_pass)

Loading MySQL configuration from encrypted custom files

library(secret); try(local_key(), silent = TRUE)

vault <- file.path(tempdir(), ".vault"); dir.create(vault); create_vault(vault)

key_dir <- file.path(system.file(package = "secret"), "user_keys")
alice_public_key <- file.path(key_dir, "alice.pub")
alice_private_key <- file.path(key_dir, "alice.pem")
add_user("alice", alice_public_key, vault = vault)

secret_to_keep <- c(password = "guest", username = "guest")
add_secret("secret_one", secret_to_keep, users = "alice", vault = vault)

secrets <- get_secret("secret_one", key = alice_private_key, vault = vault)
mysql_user <- secrets$my_username; mysql_pass <- secrets$my_password

con <- DBI::dbConnect(
    RMySQL::MySQL(), dbname = "shinydemo",
    host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
    username = mysql_user, password = mysql_pass)

Loading MySQL configuration from environment variables

library(DBI)
con <- dbConnect(
    RMySQL::MySQL(),
    dbname = "shinydemo",
    host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
    username = Sys.getenv("my_username"),
    password = Sys.getenv("my_password"))

$ cat .Renviron
my_username = "guest"
my_password = "guest"

$ docker run --env my_username=guest --env my_password=guest my_docker_image ...

$ aws ecs register-task-definition --container-definitions ...

Loading MySQL configuration from config files v1

conf <- config::get("shinydemo")
con  <- DBI::dbConnect(
    RMySQL::MySQL(),
    dbname = conf$dbname,
    host = conf$host,
    username = conf$username,
    password = conf$password)

With the below YAML config:

default:
  shinydemo:
    host: shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com
    username: guest
    password: guest
    dbname: shinydemo

Loading MySQL configuration from config files v2

conf <- config::get("shinydemo")
do.call(DBI::dbConnect, conf)

With the below YAML config:

default:
  shinydemo:
    drv: !expr RMySQL::MySQL()
    host: shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com
    username: guest
    password: guest
    dbname: shinydemo

Amazon KMS for passwords

Source: AWS Encryption SDK

Amazon KMS for 4+ kbytes data

Source: AWS Encryption SDK

Loading MySQL configuration from config files v3

conf <- config::get("shinydemo")
do.call(DBI::dbConnect, conf)

With the below YAML config:

default:
  shinydemo:
    drv: !expr RMySQL::MySQL()
    host: shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com
    username: !aws_kms ...
    password: !aws_kms ...
    dbname: shinydemo

TODO

sss

Examples

todo

Examples

todo

Installing Jenkins

1. Install Jenkins from https://pkg.jenkins.io/debian-stable/

wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
echo "deb https://pkg.jenkins.io/debian-stable binary/" | sudo tee -a /etc/apt/sources.list
sudo apt update
sudo apt install openjdk-8-jdk-headless jenkins
sudo netstat -tapen | grep java

2. Open up port 8080 in the related security group / firewall

3. Access Jenkins from your browser and finish installation

   1. Read the initial admin password:

   sudo cat /var/lib/jenkins/secrets/initialAdminPassword

   2. Proceed with installing the suggested plugins
   3. Create your first user(s)

Scheduling jobs via cron

crontab.guru

H => hash

Jenkins output

rotation

sync to cloudwatch

Jenkins plugins

todo

extended email (edit subject / body, attachement, inline images – even from docker) slack notification

https://stackoverflow.com/questions/8175912/load-multiple-packages-at-once/8176099#8176099

How it feels when you look into a new issue opened for some of your stale #rstats projects https://t.co/CvuED5eB5A pic.twitter.com/LGQC76bcOX

— Gergely Daróczi (@daroczig) December 11, 2015

code

1 + 2
ggplot(mtcars, aes(3)

Later on

https://github.com/rajgoel/reveal.js-plugins/tree/master/embed-tweet

https://twitter.com/daroczig/status/675342535267495936

https://twitter.com/daroczig/status/682966258904510464